1. Field of the Invention
The present invention is directed to a television monitor field shifter used in conjunction with two images, collected from different viewpoints and transmitted, for stereoscopic reproduction, to a single monitor. More specifically, the present invention is directed to a method of providing a stereo image of optimal depth resolution and reduced depth distortion on a single screen with regard to a selected region of interest that lies within the field of view of the image-obtaining system.
2. Brief Description of the Prior Art
It is known in the prior art to display on a single screen a "composite" image taken by a pair of video or like television cameras for the purpose of creating a stereo image. The "composite" image displayed on the video screen is created by alternately displaying the respective images taken by a left and then a right camera of a camera pair. In accordance with the past state of the art, a monitor first presents all of the odd lines of a TV image upon the screen and next presents all of the even lines. All of the odd and even lines collectively are called odd and even fields respectively. In accordance with a national standard, (NTSC Standard Television) this process is repeated thirty (30) times per second. Still in accordance with the prior art, this standard form of picture presentation is modified for three dimensional or "stereo" presentation of images on a single screen, by transmitting directly from the left camera, an image to be displayed on the screen by, e.g. the odd numbered lines, and thereafter projecting from the right camera an image to be displayed by the even numbered lines. A complete scan of the screen with odd and even fields is called a "frame" and the process of alternately displaying left and right images within a single frame is called "interlacing".
For "stereo" viewing of the interlaced left and right images a human observer must view the left image with his left eye and the right image with his right eye. One method of accomplishing this is by viewing the screen through a state-of-the-art liquid crystal stereo viewer which opens and closes light shutters in front of the eyes. The operation of the shutters in the stereo viewer is synchronized with the image being presented on the monitor so that the left eye sees the screen when the odd lines are displayed and the right eye sees the screen when the even lines are displayed.
The foregoing is accomplished in the prior art in connection with a substantially standard (cathode ray tube or the like) video screen for displaying the images taken by the left and right cameras. The left and rights fields, representative of image information, from the left and right cameras are applied to a switching box, which box interlaces the two fields to form each frame on the monitor.
Remote monitoring and control of objects, such as remote manipulation and control of equipment and robots has developed a need for an improved stereo control system. It will be readily appreciated by those skilled in the art that accurate stereo viewing of a remote site is especially important for remote manipulation of robots. Such applications may be required, for example, while working in outer space, underwater, or in nuclear facilities, to list some typical tasks.
A search of the prior art has developed several patents of interest to the application.
U.S. Pat. No. 4,647,965 to Imsand discloses a method for improving the viewing comfort of three dimensional images, including the shifting of images of objects, so that the object images on the screen are within the viewer's limit of binocular fusion. As a result of the shifting, the viewer does not see double images. Although the system of this patent is acceptable for entertainment purposes, it is not of significant value for remote control purposes, and one following the Imsand disclosure will create depth errors by visually changing an object's perceived depth.
U.S. Pat. No. 4,399,456 to Zalm suggests a device for shifting the left image in a three dimensional television display so that the viewer does not see scene information from the side edges of the screen. It is questionable whether any image shifting actually occurs, but assuming that there is image shifting, such shifting is restricted in that the left image can only be shifted to the right, and the right image can not be shifted at all. The extent of the shifting is no more than approximately six percent (6%) of the picture width.
U.S. Pat. No. 4,677,468 to Moroshita discloses a system wherein the images of two TV monitors are projected with optical lenses upon a single non-electric viewing screen. A device shifts the timing of the readout of the left eye frame memory and of the right eye frame memory with respect to each other, for the purpose of allowing the viewer to select an observation point different from that of the camera. Frames and not fields are shifted by Moroshita and there is not mention of the problem nor solution for depth distortion in this reference. Moreover, applicant's invention is directed to a TV monitor screen field shifter used to horizontally shift all possible combinational directions of two fields to be interlaced on the TV monitor screen. Moroshita does not teach or suggest applicant's invention.
For still further background to the present invention, reference is made to U.S. Pat. Nos. 3,959,580 and 3,598,032 (Chocol et al and Bohn et al).
In performing and reporting on the experiments that led to the development of this invention, the inventor has reported on some of the approaches discussed in the literature relative to stereo imaging. The report of the experimentation of which the inventor is the primary author, is entitled "Stereo Depth Distortions in Teleoperation." By Daniel B. Diner and Marika von Sydow, dated Feb. 1, 1987 JPL publication 87-1. That report ("Diner Report") is incorporated herein by reference.
As discussed in the Diner Report, camera configurations which are similar to natural human viewing conditions are called orthostereoscopic; unnaturally wide camera separation configurations are called hyperstereoscopic. In the literature on stereo imaging, some researchers advocate orthostereoscopic camera alignments, and other researchers advocate hyperstereoscopic camera alignment.
Shields, Kirkpatrick, Malone and Huggins, "Desing Parameters for a Stereoptic Television System Based on Direct Depth Perception Cues," Washington, D.C.: Proceedings of the Human Factors Society 19th Annual Meeting, (1975), pp. 423-427, found no gain in performance with hyperstereopsis on a stereo depth comparison task, and recommended orthostereopsis. This recommendation is not surprising as the depth distortion of hyperstereopsis may well have overridden the advantage of the increased depth resolution.
Grant, Meirick Polhemus, Spencer, Swain, and Tewell, "Conceptual Design Study for a Teleoperator Visual System Report," Denver, Co: Martin Marietta Corporation Report NASA CR-124273, (April 1973), found no gain in performance with hyperstereopsis on a peg-in-hole task and recommended orthostereopsis. A peg-in-hole task requires high depth percision only in a small region of the work space. The depth distortion of hyperstereopsis only becomes significant for objects which are separated horizontally. Thus the performance of the insertion of the peg into the hole should increase with the increased depth resolution of hyperstereopsis. Perhaps the depth distortions hurt the performance of the long range motions (such as moving towards the peg and moving the peg towards the hole) enough to overshadow the increase in performance of the insertions.
Upton and Strother, "Design and Flight Evaluation of a Helmet-mounted Display and Control System," In Birt, J. A. & Task, H. L. (Eds.), A symposium on visually coupled systems: Development and application (AMD-TR-73-1), Brooks Air Force Base, TX, (September, 1973), reported that hyperstereopsis greatly enhanced depth detection of camouflaged buildings from helicopter-mounted stereo camera. This result is expected. The critical point to be noted is that the accurate detection of depth is a different phenomenon from the accurate estimate of the magnitude of a true depth. Hyperstereopsis artificially magnifies the perceived magnitude of a true depth difference, making that depth difference easier to detect, but much harder to perform accurate teleoperation upon. For example, hyperstereopsis might make a one-story camouflaged building appear to be four stories tall.
Zamarian, "Use of Stereopsis in Electronic Displays: Part II--Stereoscopic Threshold Performance as a Function of System Characteristics," Douglas Aircraft Company Report MDC J7410, (December, 1976), reported that hyperstereopsis improved performance over orthostereopsis on a three-bar depth adjustment ask, using converged cameras. The three-bar depth adjustment task insures that the depth distortions will play a role in his experiment. He states, ". . . it was found that performance improved with increasing [camera] separation but at a decreasing rate of improvement." It is suspected that he was experiencing a trade-off between increased resolution and distortion.
Pepper, Cole, and Spain, "The Influence of Camera Separation and Head Movement on Perceptual Performance Under Direct and TV-displayed Condition," Proceedings of the Society for Information Display, 24, (1983), pp. 73-80, reported that hyperstereopsis improved performance on a two-bar depth adjustment task. They used parallel camera configurations, and therefore introduced no stereo depth distortions. Such results, therefore, do not compare directly to the invention presented in this application, nor do the results teach or suggest the invention.
Spain, "A Psychophysical Investigation of the Perception of Depth with Stereoscopic Television Displays," University of Hawaii doctoral dissertation, (May 1984), reported that hyperstereopsis improved performance on a two-bar depth adjustment task. He converged the cameras so that the camera convergence point was half-way between the two bars when the bars were located at equal depth. It is felt that each bar experience the same depth distortion. The net effect then would have been that the relative distortion between the two bars cancelled out. In that case, the increased stereo depth resolution of hyperstereopsis would have improved performance.
Bejczy, "Performance Evaluation of Computer-aided Manipulator Control." IEEE International Conference on Systems, Man and Cybernetics, (1976), reported surprisingly poor performance with a stereo TV viewing system of a task which required the positioning and orienting of an end-effector in an almost static visual scene. Operators were required to pick up one block and place it upon another block. Although the thrust of this work was to evaluate the effect of short-range proximity sensors in conjunction with mono and stereo camera systems on the performance of this task, the surprisingly poor performance with stereo viewing must be noted.
In reviewing the literature, it was noticed that most analyses of stereo TV viewing use small angle approximations. However, I have discovered that the actual stereo distortion of the fronto-parallel plane of convergence is such that small angle approximations obscure the relationship between this distortion and the key parameters of the camera configurations.
The above-noted prior art patents that disclose stereo systems for entertainment are aimed primarily at viewer comfort and are not useful for the critical tasks of teleoperation. The literature discussed above and the known patents applicable to teleoperation (Morishita) do not appear to have perceived that the visual depth distortion and depth resolution varies within the field of view of the cameras. Additionally the use of small angle approximations by the prior art study approaches has obscured valuable parameters concerning depth distortion and depth resolution. This invention discloses and claims horizontal field shifting for a TV monitor image and a stereo image presentation technique on a standard TV monitor which increases depth resolution while decreasing depth distortion, thus solving the resolution/distortion trade-off.